Optimization of Cross-Lingual Voice Conversion With Linguistics Losses to Reduce Foreign Accents

Cross-lingual voice conversion (XVC) transforms the speaker identity of a source to that target who speaks different language. Due intrinsic differences between languages, converted speech may carry an unwanted foreign accent. In this paper, we first investigate intelligibility and confirm performance degradation caused by accent/intelligibility issue. With goal generating native-sounding speech, paper further proposes novel training scheme with two additional linguistic losses for waveform generation: 1) frame-wise phonetic content loss derived from bottleneck features, 2) automatic recognition on characters. Experiments were conducted English Mandarin Chinese conversions. The experimental results confirmed generated sounds more natural proposed solution significantly improves intelligibility.